Skip to content

Use std::vector for varying sized buffers#1229

Merged
feldergast merged 2 commits intosstsimulator:develfrom
leekillough:use_vector
Mar 9, 2025
Merged

Use std::vector for varying sized buffers#1229
feldergast merged 2 commits intosstsimulator:develfrom
leekillough:use_vector

Conversation

@leekillough
Copy link
Contributor

@leekillough leekillough commented Mar 5, 2025

While investigating Valgrind errors, I noticed this code which is much simpler and less error-prone if std::vector<char> is used to manage a varying-size buffer.

@github-actions github-actions bot added AT: WIP Mark PR as a Work in Progress (No Autotesting Performed) AT: CLANG-FORMAT PASS and removed AT: WIP Mark PR as a Work in Progress (No Autotesting Performed) labels Mar 5, 2025
@github-actions
Copy link

github-actions bot commented Mar 5, 2025

CLANG-FORMAT TEST - PASSED

@github-actions
Copy link

github-actions bot commented Mar 5, 2025

CMAKE-FORMAT TEST - PASSED

@sst-autotester
Copy link
Contributor

sst-autotester commented Mar 5, 2025

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

@github-actions github-actions bot added AT: WIP Mark PR as a Work in Progress (No Autotesting Performed) and removed AT: WIP Mark PR as a Work in Progress (No Autotesting Performed) labels Mar 5, 2025
@github-actions
Copy link

github-actions bot commented Mar 5, 2025

CMAKE-FORMAT TEST - PASSED

@github-actions
Copy link

github-actions bot commented Mar 5, 2025

CLANG-FORMAT TEST - PASSED

@sst-autotester
Copy link
Contributor

sst-autotester commented Mar 5, 2025

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

@leekillough
Copy link
Contributor Author

leekillough commented Mar 6, 2025

There are valgrind uninitialized memory errors which are not fixed by this change which was motivated by analyzing these valgrind errors, but the change is wanted nonetheless, since it makes the code simpler.

==213066== Syscall param writev(vector[1]) points to uninitialised byte(s)
==213066==    at 0x5408034: writev (writev.c:26)
==213066==    by 0x513EC71: std::__basic_file<char>::xsputn_2(char const*, long, char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.32)
==213066==    by 0x518332B: std::basic_filebuf<char, std::char_traits<char> >::xsputn(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.32)
==213066==    by 0x51AC9A5: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.32)
==213066==    by 0x319D5A: SST::Simulation_impl::checkpoint(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (simulation.cc:1729)
==213066==    by 0x270EEB: SST::CheckpointAction::createCheckpoint(SST::Simulation_impl*) (checkpointAction.cc:130)
==213066==    by 0x271A1C: SST::CheckpointAction::execute() (checkpointAction.cc:79)
==213066==    by 0x317187: SST::Simulation_impl::run() (simulation.cc:877)
==213066==    by 0x236C60: start_simulation(unsigned int, SimThreadInfo_t&, SST::Core::ThreadSafe::Barrier&) (main.cc:601)
==213066==    by 0x217F73: main (main.cc:1109)
==213066==  Address 0x6c97042 is 98 bytes inside a block of size 16,912 alloc'd
==213066==    at 0x4859047: operator new(unsigned long) (vg_replace_malloc.c:472)
==213066==    by 0x31DA4B: allocate (new_allocator.h:137)
==213066==    by 0x31DA4B: allocate (alloc_traits.h:464)
==213066==    by 0x31DA4B: _M_allocate (stl_vector.h:378)
==213066==    by 0x31DA4B: std::vector<char, std::allocator<char> >::_M_default_append(unsigned long) (vector.tcc:657)
==213066==    by 0x319E80: resize (stl_vector.h:1011)
==213066==    by 0x319E80: SST::Simulation_impl::checkpoint(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (simulation.cc:1722)
==213066==    by 0x270EEB: SST::CheckpointAction::createCheckpoint(SST::Simulation_impl*) (checkpointAction.cc:130)
==213066==    by 0x271A1C: SST::CheckpointAction::execute() (checkpointAction.cc:79)
==213066==    by 0x317187: SST::Simulation_impl::run() (simulation.cc:877)
==213066==    by 0x236C60: start_simulation(unsigned int, SimThreadInfo_t&, SST::Core::ThreadSafe::Barrier&) (main.cc:601)
==213066==    by 0x217F73: main (main.cc:1109)
==213066==  Uninitialised value was created by a heap allocation
==213066==    at 0x4859047: operator new(unsigned long) (vg_replace_malloc.c:472)
==213066==    by 0x2966DE: SST::ConfigGraph::addComponent(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (configGraph.cc:784)
==213066==    by 0x3DD16E: addComponent (pymodel.h:94)
==213066==    by 0x3DD16E: compInit(ComponentPy_t*, _object*, _object*) (pymodel_comp.cc:95)
==213066==    by 0x4A7E431: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.11.so.1.0)
==213066==    by 0x4A186C0: _PyObject_MakeTpCall (in /usr/lib/x86_64-linux-gnu/libpython3.11.so.1.0)
==213066==    by 0x49ABBB6: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.11.so.1.0)
==213066==    by 0x4C6E13B: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.11.so.1.0)
==213066==    by 0x4B069F4: PyEval_EvalCode (in /usr/lib/x86_64-linux-gnu/libpython3.11.so.1.0)
==213066==    by 0x4B55E28: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.11.so.1.0)
==213066==    by 0x4B567E5: _PyRun_SimpleFileObject (in /usr/lib/x86_64-linux-gnu/libpython3.11.so.1.0)
==213066==    by 0x4B56DBF: _PyRun_AnyFileObject (in /usr/lib/x86_64-linux-gnu/libpython3.11.so.1.0)
==213066==    by 0x4B56E7C: PyRun_AnyFileExFlags (in /usr/lib/x86_64-linux-gnu/libpython3.11.so.1.0)
==213066==

With this PR's changes, the use of std::vector instead of new char[] makes the memory initialized to 0 upon creation and resizing (the same effect could have been done by adding {} to the end of new char[]), but it does not remove the uninitialized memory error, which seems to mean that the uninitialized data is copied from somewhere else to buffer or size is wrong. valgrind flags the uninitialized memory read as occurring in:

        fs.write(&buffer[0], size);

in the part of the loop:

    // Serialize component blobs individually
    for ( auto comp = compInfoMap.begin(); comp != compInfoMap.end(); comp++ ) {
        ser.start_sizing();
        ComponentInfo* compinfo = *comp;
        ser&           compinfo;
        size = ser.size();
        buffer.resize(size);

        ser.start_packing(&buffer[0], size);
        ser& compinfo;

        component_blob_offsets_.emplace_back(compinfo->id_, offset);
        fs.write(reinterpret_cast<const char*>(&size), sizeof(size));
        fs.write(&buffer[0], size);
        offset += (sizeof(size) + size);
    }

The buffer.resize(size) call should initialize all new elements to 0 if size increases, but the ser& compinfo should initialize all size elements, if the size reported by the ser.start_sizing() ... size = ser.size() sequence is working correctly (which it might not be).

@sst-autotester
Copy link
Contributor

sst-autotester commented Mar 8, 2025

Status Flag 'Pre-Test Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED by label AT: PRE-TEST INSPECTED! Autotester is Removing Label; this inspection will remain valid until a new commit to source branch is performed.

@sst-autotester
Copy link
Contributor

sst-autotester commented Mar 8, 2025

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements

  • Build Num: 1856
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements_MR-2

  • Build Num: 1812
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements_MT-2

  • Build Num: 1811
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-macro_withsstcore

  • Build Num: 809
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-core_Make-Dist

  • Build Num: 662
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_Clang-Format_sst-core

  • Build Num: 616
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_OSX-14-XC15-ARM2_OMPI-4.1.6_PY3.10_sst-elements

  • Build Num: 379
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_OSX-14-XC15-ARM2_OMPI-4.1.6_PY3.10_sst-macro_withsstcore

  • Build Num: 234
  • Status: STARTED

Using Repos:

Repo: CORE (leekillough/sst-core)
  • Branch: use_vector
  • SHA: 19b6a50
  • Mode: TEST_REPO
Repo: SQE (sstsimulator/sst-sqe)
  • Branch: devel
  • SHA: fc3830fbe74ce06cebb6b6838c78777a6855707a
  • Mode: SUPPORT_REPO
Repo: ELEMENTS (sstsimulator/sst-elements)
  • Branch: devel
  • SHA: de2c6802a84ae1701a768d7387fc207adc867056
  • Mode: SUPPORT_REPO
Repo: MACRO (sstsimulator/sst-macro)
  • Branch: devel
  • SHA: 42e85e1689d473c65fdbcc008ce57fd53fe80865
  • Mode: SUPPORT_REPO

Pull Request Author: leekillough

@sst-autotester
Copy link
Contributor

sst-autotester commented Mar 9, 2025

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements

  • Build Num: 1856
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements_MR-2

  • Build Num: 1812
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements_MT-2

  • Build Num: 1811
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-macro_withsstcore

  • Build Num: 809
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-core_Make-Dist

  • Build Num: 662
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_Clang-Format_sst-core

  • Build Num: 616
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_OSX-14-XC15-ARM2_OMPI-4.1.6_PY3.10_sst-elements

  • Build Num: 379
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_OSX-14-XC15-ARM2_OMPI-4.1.6_PY3.10_sst-macro_withsstcore

  • Build Num: 234
  • Status: PASSED

@sst-autotester
Copy link
Contributor

sst-autotester commented Mar 9, 2025

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ feldergast ]!

@sst-autotester
Copy link
Contributor

sst-autotester commented Mar 9, 2025

Status Flag 'Pull Request AutoTester' - Pull Request MUST BE MERGED MANUALLY BY Project Team - This Repo does not support Automerge

@feldergast feldergast merged commit bf35870 into sstsimulator:devel Mar 9, 2025
7 checks passed
kpgriesser pushed a commit to tactcomplabs/sst-core that referenced this pull request Mar 14, 2025
@leekillough
Use std::vector for varying sized buffers (sstsimulator#1229)
2de4955
@leekillough leekillough deleted the use_vector branch March 31, 2025 04:06
@berquist berquist added this to the SST V15.0.0 milestone Apr 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@feldergast feldergast feldergast approved these changes

Assignees

No one assigned

Labels

AT: CLANG-FORMAT PASS AT: CMAKE-FORMAT PASS

Projects

None yet

Milestone

SST V15.0.0

Development

Successfully merging this pull request may close these issues.

4 participants

@leekillough @sst-autotester @feldergast @berquist